Analyzing Quantitative Models

This article presents a framework for the evaluation of quantitative models. The framework is both simple and realistic and could be used profitably by most organizations. It incorporates not only internal accounting data but also the human elements of bias or antipathy toward the models on the part of company employees using them, which might tend to distort an internal assessment and even the capability of the model itself. Comments Postprint version. Published in Journal of Marketing, Volume 38, Issue 2, April 1974, pages 61-66. Publisher URL: http://www.ama.org/pubs/jm/ This journal article is available at ScholarlyCommons: http://repository.upenn.edu/marketing_papers/96 Analyzing Quantitative Models J. Scott Armstrong and Alan C. Shapiro Reprinted with permission from Journal of Marketing, Vol. 38, No. 2, (1974), 61-66 How can a potential user distinguish between a quantitative model that may be of some real value and one that is not? The model builder rarely provides much help, since most are advocates of their own work and tend to lose their objectivity toward the model. Therefore, an independent evaluation is necessary to judge the true usefulness of the model. This article presents a framework for the evaluation of quantitative models. The framework is both simple and realistic and could be used profitably by most organizations. It incorporates not only internal accounting data but also the human elements of bias or antipathy toward the models on the part of company employees using them, which might tend to distort an internal assessment and even the capability of the model itself. The framework will be discussed in terms of its component concepts and their interrelationships. In the course of this discussion, illustration will be made of how this framework was used in an actual situation to evaluate a particular set of models, which will be referred to here as the FAITH models. The proponents of these models, which are currently being used by some of the largest corporations in the United States, claim they are useful for predicting the outcomes of various marketing strategies. For example, they may be used to predict market share when changes are made in prices, in product line, or in advertising. The authors do not wish to imply that the example they have selected is typical of all quantitative models. In fact, it was selected as an illustration because it contains a number of serious defects. The FAITH models use concepts from the physical sciences to deduce "fundamental laws of consumer behavior." Consumer behavior in a wide variety of situations can supposedly be predicted by first examining the pattern of brand switching among products to determine which products compete with one another. Given this description of the market, predictions may then be made by examining the relationship between the variable of interest (e.g., market share) and one or more key market variables (e.g., planned advertising expenditures by the firm). The models are very simple in terms of the data requirements; however, the nature of the relationships among the key variables tends to be very complex. The authors conducted an evaluation of the FAITH models for a large beverageproducing company, which will be referred to here under the fictitious name National Beverage, Inc. It should be emphasized that these models are well known and that National Beverage is quite sophisticated in the application of quantitative marketing techniques. National Beverage had been subscribing to the FAITH models for almost six years but had actually made only modest use of them up to the time that this study was undertaken. Their six years of "evaluation" led top management to a very favorable opinion of the FAITH models. However, this evaluation was not carried out in a systematic way (there was no master plan for the evaluation); it was incomplete (certain key aspects had not been examined); and it was not done in an explicit way ("We didn't write up any reports on this"). The framework discussed in this article is designed to provide a systematic, comprehensive, and explicit evaluation. The conclusion yielded by the use of this framework contrasted sharply with that yielded by the previous evaluation. A Framework for Evaluation The usefulness of a quantitative model depends on both "acceptability" and "quality." Acceptability refers to approval by those in the organization who would actually use the model, while quality refers to the ability to provide better predictions or decisions. A model must score well on both characteristics if it is to be judged useful. A high-quality model that is not accepted is of no value. Usually, some trade-offs must be made between quality and acceptability. For all practical purposes, quality and acceptability must be viewed in relative terms. That is, these concepts can only be examined by a comparison of alternative models. A model is said to be "good" if it is better than alternative models. Among the alternative models would certainly be included the way in which these predictions or decisions are currently being made. In most cases, the current method is based entirely upon the judgment of a manager. Woolsey, for example, examined four applications of complex computer models and claimed that they were inferior to the decisions currently being made subjectively. In one case, for example, two "little old ladies" did far better than a complex and expensive computer model. Quality and acceptability are characteristics that may depend not only upon the model but also upon the situation. The fact that the model worked for one company does not necessarily mean that it will also work for National Beverage. This is yet another reason why the potential user should carry out his own examination of the model. Finally, it is important that the evaluation of both quality and acceptability be carried out by an unbiased evaluator. The potential user is generally unbiased prior to the purchase of a given. Once someone in the organization has become committed to the model (as happened in National Beverage's use of FAITH), internal objectivity is difficult. It is also important, in this case, to avoid the use of an outside evaluator who has a competitive model to promote. These concepts of acceptability and quality are examined in greater detail below. Evaluating Acceptability One must consider both whether the model will be used and, if used, how it will be used. High quality solutions are often misused and may create other problems (e.g., make the organization more resistant to the introduction of further changes). On the other hand, lowquality solutions are often high in acceptability. 1 R. E. D. Woolsey, "A Candle to Saint Jude, or Four Real World Applications of Integer Programming," Interfaces, Vol. 2 (February 1972), pp. 20-27. In the framework presented here, the authors propose that the evaluation of acceptability be carried out by judging the model through the eyes of the user. In particular, the following should be examined: 1. Perceived quality – that is, the user's perception of the value of the model to the organization 2. Perceived personal value – that is, the user's perception of the benefits (or costs) of the model to his own career. Interviews or questionnaires offer the most direct way to assess user perceptions. In some cases, however, there may also be indirect or "unobtrusive" measures. In evaluating the acceptability of the FAITH models, a series of group and individual interviews was conducted. Members of the user group were sampled in an attempt to include all levels within the marketing division, as well as related staff groups. Two interviewers sat in on each interview and each took notes separately so that the reliability of this information could be maximized. During the interviews, interviewers were careful to avoid evaluating what was said, since this might have biased the replies. The highlights of this effort are summarized below. Perceived Quality of the FAITH Models A substantial number of users did not have confidence in the assumptions of the FAITH models. Numerous comments were made along the lines of, "They make a lot of assumptions open to question." It is interesting to note that the advocates of FAITH ask people to "suspend belief" when they first explain the assumptions. The company was unable to locate a single user who claimed to have an adequate understanding of how FAITH worked! Typical comments were: "No one can explain FAITH to me." "There's an inability to communicate." "I don't know how FAITH works." Perceptions as to how well the FAITH models predicted were mixed. The top two levels in National Beverage felt that FAITH provided better predictions than those being made by the product managers. The middle and lower levels felt just the opposite—partially, it would seem, because they had confidence in their own judgment. Comments by these latter managers included: "FAITH can't be wrong; if their estimates are off they [FAITH advocates] claim it's because the input data were not right." Perceived Personal Value to Users of FAITH On the question of personal value, there was again a split in opinion by organizational level. The higher levels felt that the FAITH models would be of value to them and would give a greater sense of control over decision-making. On the other hand, the middle and lower levels felt that this was another attempt by higher management to reduce the influence of the lower levels on decision making. The low level of acceptability of FAITH also showed up in other ways. For example, when the middle and lower levels of management of National Beverage met for an all-day planning session, not a single reference was made to FAITH; yet FAITH was then regarded by top management as the foremost quantitative planning model in the company. Because marketing managers had been told by top management to use the FAITH models, another problem arose. Middle and lower level managers said that it was common to change the inputs to the FAITH model until they found a result that agreed with their own decision. This version was then presented to top management. In other words, these managers were misusing the models.